fix(databricks): a schema the contract stops using is destroyed, not left failing the apply - #873

What

A Databricks table that moves to another schema, or a schema whose last expose is removed, no longer leaves fluid apply failing at OpenTofu's destroy of the old schema ("Schema is not empty"). The product's metric views and its column-mask and row-filter functions there are no OpenTofu resource, and Unity Catalog refuses to delete a schema that holds anything, so until now someone had to drop them by hand. This was the documented "known limit" of #865 and of the masks' functions.

How

• Which schemas. From the saved plan, never from a catalog search: each databricks_schema the plan only deletes and the contract no longer names (databricks_vacated.vacated). Only schemas this environment's own OpenTofu state holds are looked in, so another environment of the product (an overlay that changes the schema, same catalog, same identity) keeps its objects.
• Before tofu apply (new plugin hook prepare_destroys, after pre_apply): drop this product's metric views (fluid.product_id, owned by the deploy identity) whose table the plan destroys or is gone, and fluid_rf_* / fluid_mask_* functions the deploy identity owns that no table of the schema uses.
• A function a table still uses is never dropped while the table exists. Unity Catalog names a mask's or row filter's function in the table (SET MASK <function name>), so dropping it early would leave the table reading through whatever function someone with CREATE FUNCTION on the schema then made under that name.
• The resume. OpenTofu destroys a table before its schema inside one apply, so a schema whose tables had masks still fails there. The engine then calls the new unblock_destroys hook: the tables are gone, so their functions are dropped by the same rules, and a second plan is applied only if it holds nothing but those schema deletes (the data-loss gate already passed them). Anything else (an error a failed delete does not account for, a plugin that clears nothing, a second plan with any other change) leaves the apply's own failure as before.
• Anything else the schema holds (a kept table, another product's view, an object another identity owns) is named, with what to do.
• --mode dry-run prints the DROP VIEW before the apply and the DROP FUNCTIONs that wait for the tables, and runs none.
• The Unity Catalog stand-in now refuses to delete a schema that holds a function, view or volume, as Unity Catalog does (a function measured live on 2026-11-03, HONESTLY_TESTED.md).

Engine plumbing: runner.planned_deletes / runner.errored_resources (sharing one event iterator with planned_removals), _apply_saved_plan / _applied out of _apply_via_opentofu, and one _run_hook for the optional plugin hooks.

Borrow before build

No prior art for clearing objects outside state between a child's and its parent's destroy. force_destroy would delete data not in state; OpenTofu -exclude needs 1.9 (the floor is 1.6.0); a destroy-time provisioner never runs once its resource block is removed (hashicorp/terraform#13549); Databricks Asset Bundles have no such hook (databricks/cli#1848). Reused the repo's own seams: _schema_views/_drop_stale checks, drop_orphan_functions' name shapes and owner check, the engine's optional-hook pattern.

Validation

• Real run (the fluid CLI, tofu 1.13 and databricks/databricks 1.140.0 against the in-process Unity Catalog stand-in): a table with a metric view, a mask and a row filter moves from main.sales to main.other. --mode dry-run prints the DROP VIEW before the apply and the DROP FUNCTIONs after the table's destroy, and runs none; the apply exits 0, prints tofu plan (the destroys that waited): +0 ~0 -1 and tofu apply complete: +11 ~0 -11, and main.sales is gone with nothing left in it. Removing a schema's last expose does the same.
• New tests: tests/iac/test_iac_databricks_vacated.py (14, the rules against the stand-in), tests/iac/test_iac_apply_engine_destroys.py (24, the engine with tofu stubbed), 2 fake e2e in tests/iac/test_iac_databricks_metrics_fake_e2e.py (in the existing Stage 1 Databricks lane).
• Gates: ruff and black 24.10.0 whole tree clean, dead-code gate, license headers, product guardrail, import-linter (6 kept), mypy --strict on the new module. tests/iac + tests/cli (non-integration): 7121 passed, 2 failed in tests/cli/test_lazy_core_commands.py, the known shared-venv baseline that fails on any tree. All Databricks fake e2e tiers serially: 61 passed, 18 skipped (need databricks-sdk), 2 failed with the stand-in's write: broken pipe and passed on re-run. That flake is the stand-in's (it answers HTTP/1.0 and closes every connection): in an interleaved A/B of the data quality tier at equal load, origin/main failed 2 of 4 runs and this branch 1 of 4. Full suite once: 35,691 passed; the 32 failures and 2 errors are the known shared-venv baselines (moto tiers needing a provider download, dotenv_dlt, duckdb sandbox, lazy-core imports, compat gate, bashlex collection) plus 3 stand-in transport flakes under -n 6.
• ripwire --quality-delta (branch diff on a fresh origin/main tree): the rows on touched code are DatabricksIacPlugin +25 lines (the two hooks), short-horizon churn of the refactored hook runners, test-helper duplication, and a dead-code false positive (MetricViews.plan_actions is called at databricks.py:2933). Refactored during the branch: one _run_hook for the optional plugin hooks, _apply_saved_plan / _applied out of _apply_via_opentofu, the plan parsing in runner.

Reviews

• Security review (Opus judge over the branch diff): no HIGH or MEDIUM finding. The one assumption it named, a bound function that the Tables API names without its schema, is closed (149043f3).
• World-class inspection (Diff mode): PASS. Delivery review found no P0-P2 and judged the resume design sound; its P3 (a stale test count) is fixed. UX of the changed CLI output scored 9/10 on the move, a schema holding foreign objects, and no client; two headings worded for one case are fixed.
• Defect scan (Diff mode, 3 hunters, 1 skeptic): 2 REAL_BUG, both Low, fixed in fd5f6644. (1) A metric view definition PyYAML cannot load (an impossible date is a ValueError), or a row of an odd shape, escaped as a traceback before tofu apply. (2) The product id came from the compiled views, which are empty when no Databricks expose is left. Refuted but hardened anyway: the resume now runs only for the destroys the plugin cleared (it returns their addresses), and metric views are also found through table-summaries.

Commits: the stand-in fidelity fix, the change, then three review fixes (each its own commit). Rebased on origin/main 1e20c739.

Not done here

• Live Databricks run (Free Edition needs a founder go): that Unity Catalog deletes the schema once its functions are dropped is inferred from the 2026-11-03 measurement, not observed.
• forge_docs has no metric view page yet; when one is written it should describe this (the behaviour lives in the module docstrings, CHANGELOG and HONESTLY_TESTED).
• The stand-in's HTTP/1.0 transport flake (a provider write to a connection the stand-in closed) predates this branch; its own tier retries one destroy for it.